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Model of a Typical Component Configuration Employing the Invention 

(57) Abstract: An apparatus and method for controlling the display, transfer and processing of information in computer systems 
(13), wherein a user aims a pointer (14) at a surface (11), such as a wall or whiteboard, which contains projected or written informa- 
tion, in order to activate predetermined operations, related to that information. A stream of digital images from the image sensor (12) 
is analyzed by the computer (13), and the appearance of the visual input along with its location are interpreted in order to perform 
the operation that was intended by the user. 
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AMENDED CLAIMS 

[received by the lnternati nal Bureau on 2 October 2000 (02.10.00); 
original claim 1 amended; new claims 2-53 added (5 pages)] 

1 . A method of controlling the display, transfer and processing of information in computer systems, said method 
comprising the step of pointing at a surface which contains information to activate predetermined computer 
operations relating to that information. 

2. A method of controlling the display, transfer and processing of information in a computer system, said method 
comprising the steps of: 

directing visual stimuli emulating a predetermined computer task on a display surface; 
capturing an image of said display surface comprising said user user directed visual stimuli; 
processing said captured image to detect said user directed visual stimuli; and 
executing said predetermined computer task. 

3. A method as in claim 2 wherein said user directed visual stimuli emulates mouse operations. 

4. A method as in claim 2 whercm said display surface further includes information that could be falsely 
interpreted as said user directed visual stimuli and wherein said detection step also includes excluding said 
informanon. 

5. A method as in claim 2 wherein said capturing step further includes acquiring a sequence of images of said 
display surface and wherein said processing step further includes analyzing said sequence of images to detect 
said user directed visual stimuli. 

6. A method as in claim 2 further comprising the step of screening said captured image to detect suspected user 
directed visual stimuli. 

7. A method as in claim 6 wherein said screening step includes sub-sampling said captured image thereby 
providing a quick screening process. 

8. A method as in claim 2 further comprising the step of preventing false activation of a computer task by 
analyzing the region of said captured image neighboring said user directed visual stimuli to exclude visual 
stimuli that could be interpreted as said user directed visual stimuli. 

9. A method as in claim 8 wherein said step of preventing raise activation of a computer task includes identifying 
a recognizable component in the neighborhood of said user directed visual stimuli ox analyzing other additional 
visual stimuli in the neighborhood of said user directed visual stimuli to exclude visual stimuli that could be 
interpreted as said user directed visual stimuli. 

10. A method as in claim 2 wherein the step of directing visual stimuli on said display surface is done by a pointing 
device having a recognizable component attached thereto. 

1 1 . A method as in claim 10 wherein said recognizable component is spherical in shape. 

12. A method as in claim 10 further comprising the step of detecting the orientation of said pointing device by 
identifying said recognizable c o mp onent. 
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13. a method as in claim 2 farther comprising the step f transmitting signals representing said user directed visual 
stimuli to another location. 

14. A method as in claim 12 further comprising the step of transmitting signals representing said user directed 
visual stimuli and said orientation of said pointing device to another location. 

15. An apparatus for controlling the display, transfer and processing of information in a computer system, said 
apparatus comprising: 

a pointing device for directing visual stimuli on a display surface; said visual stimuli to emulate a predetermined 
computer operation; 

a device configured to capture an image of said display surface comprising said user directed visual stimuli; and 

a computer configured to 1) process said captured image to detect said user directed visual stimuli and 2) 
execute said predetermined computer task. 

16. An apparatus as in claim 15 wherein said user directed visual stimuli emulates mouse operations. 

17 An apparatus as in claim 15 wherein said display surface further includes information that could be falsely 

" interpreted as said user directed visual stimuli and wherein said computer is configured to also exclude said 
information. 

18 An apparatus as in claim 15 wherein said device is configured to acquire a sequence of images of said display 

" surface and wherein said computer is configured to analyze said sequence of images to detect said user directed 
visual stimuli. 

19. An apparatus as in claim 15 wherein said computer is also configured to screen said captured image to detect 
suspected user visual stimuli. 

20. An apparatus as in claim 19 wherein said screening of said captured image includes sub-sampling said captured 
image thereby providing a quick screening process. 

21 An apparatus as in claim 15 wherein said computer is further configured to prevent false activation of a 
computer task by analyzing the region of said captured image neighboring said user directed visual stimuli to 
exclude visual stimuli that could be interpreted as said user directed visual sumuh. 

22 An apparatus as in claim 21 wherein said computer is configured to prevent false activation of a ccrnputer task 
' by id^fying a recognizable component in tbe neighborhood of said user directed visual stimuli or by 

analyzing other additional visual stimuli in the neighborhood of said user directed visual stimuli to exclude 
visual stimuli that could be interpreted as said user directed visual stimuli. 

23. An apparatus as in claim 15 wherein said pointing device includes a recognizable component attached thereto. 

24. An apparatus as in claim 23 wherein said recognizable component is spherical in shape. 
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25. An apparatus as in claim 23 wherein said computer is also configured to detect the orientation of said pointing 
device by identifying said recognizable component 

26. An apparatus as in claim 15 wherein said computer includes a transmission interface and wherein said computer 
is also configured to transmit signals representing said user directed visual stimuli to another location. 

27 An apparatus as in claim 25 said computer includes a transmission interface and wherein said computer is also 
configured to transmit signals representing said user directed visual stimuli and said orientation of said pointing 
device to another location. 



28. A method controlling the display, transfer and processing of information in a computer system, said method 
comprising the steps of: 

projecting a video image on a surface capable of displaying said video image; 

directing visual stimuli emulating a predetermined computer task on said projected video image; 

capturing an image of said projected video image comprising said user directed visual stimuli; 

processing said captured image to detect said user directed visual stimuli; and 

executing said predetermined computer task. 

29. A method of claim 28 wherein said user directed visual stimuli emulates mouse operations. 

30. A method as in claim 28 wherein said projected video image further includes information that could be felsely 
interpreted as visual stimuli and wherein said detection step also includes excluding said information. 

31 A method as in claim 28 wherein said capturing step further includes acquiring a sequence of images of said 
projected video image and wherein said processing step further includes analyzing said sequence of images to 
detect said user directed visual stimuli 

32. A method as in claim 28 further comprising the step of screening said captured image to detect suspected user 
visual stimuli. 

33. A method as in claim 32 wherein said screening step includes sub-sampling said captured image thereby 
providing a quick screening process. 

34. A method as in claim 28 further comprising the step of preventing false activation of a computer task by 
analyzing Ac region of said captured image neighboring said user directed visual stimuli to exclude visual 
stimuli that could be interpreted as said user directed visual stimuli. 

35. A method as in claim 34 wherein said step of preventing felse activation of a computer task includes identifying 
a recognizable component in the neighborhood of said user directed visual stimuli or analyzing other additional 
visual stimuli in the neighborhood of said user directed visual stimuli to exclude visual stimuli that could be 
interpreted as said user directed visual s timu l i. 

36. A method as in claim 28 wherein the step of directing visual stimuli on said display surface is done by a 
pointing device having a recognizable component attached thereto. 

1009H6-5 
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37. A method as in claim 36 wherein said recognizable component is spherical in shape, 

38. A method as in claim 36 further comprising the step of detecting the orientation of said pointing device by 
identifying said recognizable component 

39. A method as in claim 28 further comprising the step of transmitting signals representing said user directed 
visual stimuli to another location. 

40. A method as in claim 38 further comprising the step of transmitting signals representing said user directed 
visual stimuli and said orientation of said pointing device to another location. 

41. An apparatus for controlling the display, transfer and processing of information in a computer system, said 
apparatus c ompr i sin g: 

a device for projecting a video image on a surface capable of displaying said projected video image; 

a pointing device for directing visual stimuli on said video image; said visual stimuli to emulate a 
predetermined computer operation; 

a device configured to capture an image of said projected video image comprising said user directed visual 
stimuli; and 

a computer configured to 1) process said captured image to detect said user directed visual stimuli and 2) 
execute said predetermined computer task. 

42. An apparatus as in claim 41 wherein said user directed visual stimuli prmilatfts mouse operations. 

43. An apparatus as in claim 4] wherein said display surface further includes information that could be falsely 
interpreted as visual stimuli and wherein said computer is configured to also exclude said information. 

44. An apparatus as in claim 4 1 wherein said device is configured to acquire a sequence of images of said projected 
video image and wherein said computer is configured to analyze said sequence of images to detect said user 
directed visual stimuli. 

45. An apparatus as in claim 41 wherein said computer is also configured to screen said captured image to detect 
suspected user visual stimuli. 

46. An apparatus as in claim 45 wherein said screening of said captured image includes sub-sampling said captured 
image thereby providing a quick screening process. 

47. An apparatus as in claim 41 wherein said computer is further configured to prevent false activation of a 
computer task by analyzing the region of said captured image neighboring said user directed visual stimuli to 
exclude visual stimuli that could be interpreted as said user directed visual stimuli. 

48. An apparatus as in claim 47 wherein said computer is configured to prevent false activation of a computer task 
by identifying a recognizable component in the neighborhood of said user directed visual stimuli or by 
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analyzing other additional visual stimuli in the neighborhood f said user directed visual stimuli to exclude 
visual stimuli that could be interpreted as said user directed visual stimuli. 

49. An apparatus as in claim 41 wherein said pointing device includes a recognizable component attached thereto. 

50. An apparatus as in claim 49 wherein said recognizable component is spherical in shape. 

5L An apparatus as in claim 41 wherein said computer is also configured to detect the orientation of said pointing 
device by identifying said recognizable component 

52. An apparatus as in claim 41 wherein said computer includes a transmission interlace and wherem said computer 
is also configured to transmit signals representing said user directed visual stimuli to another location. 

53. An apparatus as in claim 5 1 said computer includes a transmission interface and wherein said computer is also 
configured to transmit si gnals representing said user directed visual stimuli and said orientation of said pointing 
device to another location. 
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Method and Apparatus f r Visual P ointing and Computer Control 

contents of the file in a word-processing or editing program. In this example, the current 
invention provides the ability to identify the object to be acted upon and the impetus to 
do so, while another technology determines which specific action should be performed. 
The prior art includes the "Cyclops Interactive Pointer" system sold by Proxima Corp., San 
Diego, CA, that is based on inventions disclosed under several United States Patents: 5,504,501 
(Hauck et. a/.), 5,515,079 (Hauck), and 5,502,459 (Marshall et. al). This apparatus emulates 
computer-mouse operation using a hand-held light-generating device. The invention was designed 
for use with liquid crystal display (LCD) panels and some specific models of LCD projectors 
manufactured by Proxima Corp. It is a hardware solution that integrates various interconnected 
components, including a CCD image sensor, timing generator, analog comparator, digital-to-analog 
.converter, a preprogrammed microprocessor, cabling and other components into a single unit that 
outputs mouse control commands to an attached computer. 

As detailed in patent 5,515,079 (columns 7 through 10) the best mode for carrying but the 
Proxima invention uses a signal-processing unit that comprises an analog to digital (A/D) 
comparator network, a synchronizing network and a position detector. The A/D comparator network 
comprises a conventional video amplifier (emitter), analog comparator LM319A and digital to 
analog converter AD557. The synchronizing network is built from a Sanyo LC9904A timing 
generator, a Sanyo LB8904 high voltage drive unit and a conventional crystal resonator. The 
position detector comprises an Intel 8051 microprocessor, several logic devices including counters, 
inverters, and latches (74HC393, 74HC374, 74HC14) and connectors and cabling interfaces to a 
computer. The signal processing unit produces signals that control the image sensing operation of 
a Sanyo LC9943 charge coupled device (CCD) (see column 4, line 65 through column 5, line 9). 
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As the CCD scans the scene it passes each pixel's intensity value to the signal-processing 
unit for analysis. The latter determines on a pixel-by-pixel basis whether the pixel can be associated 
with a viewed light spot produced by the light-generating device (preferably a "Pocket Laser" sold 
by Navitar, see column 4, line 3 1). This deterrnination is made solely based on the intensity of the 
pixel relative to a predetermined threshold value. 

In contrast to the above description the current invention uses off-the-shelf video camera and 
frame-grabber technology (required for some cameras) to reproduce full-frame digital video images 
inside the computer's RAM (random access memory). The innovation of the invention is in the 
software that performs the image processing analysis. Rather than operating on a sequential scan 
of pixels one at a time the invention filters the entire digital image or selected regions of the image 
.at once by directly accessing any desired subset of pixel values which are directly accessible from 
the memory buffer that stores the image with all pertinent image information (including pixel 
intensities and color) of a "snapshot," or frame, of the visual scene.. See Figure 4 for an example of 
such an image region (color information is not shown). Some information is retained over time, i.e. 
during multiple frames (in some cases using more abstracted representations) in additional memory 
buffers. The software may also utilize additional, related information available from within the 
computer system. For example, the pixel values of the currently projected Computer Display Image 
are easily obtained for analysis and comparison to the sensed values of the Viewed Image. 

These structural differences from the prior art allow for much greater flexibility and power 
in the steps of analysis implemented. For example, a variety of pattern recognition algorithms are 
used to analyze and classify viewed shapes. Color values are used to help discriminate between the 
image of a targeted object or a human hand and the image background. Motion can be detected and 
isolated by analyzing information retained from a sequence of image frames. Thus, rather than 
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basing the detection of observed events solely on intensity thresholds, the current invention uses 
much more information (spatial, temporal and color) that are filtered by a large number of processing 
methods in order to determine what event, if any, is taking place. 

An important point in this regard relates to the robustness and reliability of the invention. 
Whereas the prior art provides a subset of the functionality described herein (specifically mouse 
emulation) it does so with significant restrictions. Specifically, using less image information (i.e., 
intensity values only and/or one pixel at a time) implies, in general, reduced reliability. It is easy 
to produce scenarios in which the inventions described in the prior art will either not respond to the 
light stimulus as expected or will induce false triggering by generating events that were not intended 
by the user. Apparently, the prior art can be expected to be reliable (with a small rate of errors) only 
-when used in certain environments and configurations. For example, the Proxima inventions work 
with LCD panels, preferably model A480SC from Computer Associates Corp. (column 4, line 57). 
The Cyclops product works with several LCD projector models manufactured by Proxima (with 
built-in CCD and signal processing hardware) such as the 2710 model. These projection devices 
produce a beam of light that has relatively weak intensity. LCD panels, a now outdated technology, 
work in conjunction with overhead projectors, which are also typically very weak. Because they are 
so weak, when employing such devices users typically dim the room lighting or turn it off 
completely so that details may be discerned in the projected images. Current projector technology 
produce beams of light that are many times brighter. This allows them to be used comfortably in 
well-lit rooms. 

The significantly higher intensity of projected light and the addition of ambient room light 
and its reflections is incompatible with the restrictions of the prior art devices. Under these 
conditions it becomes impossible, in general, to determine threshold intensity values to discriminate 
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spots of light because many areas of the projected image may exceed any preset threshold. 
Similarly, reflections from jewelry, belt-buckles or other objects may also produce transient spots 
of high intensity that could be falsely detected as mouse events. These problems can be partly 
oived in some cases by special adaptations made to the image-sensing device. For example, patent 
5,504,501 discloses use of a "band pass filter disposed over the lens of the device" (column 4 ; line 
30). This filter "limits the range of wavelengths of light permitted to be sensed by the device." In 
effect, the filter reduces the chances that other light sources will confuse the device since it is tuned 
to the (typically red) light of the light-generating device in use. This and similar solutions improve 
the robustness of the prior art devices, but they cannot completely overcome the problems described 
above. Furthermore, the disclosed inventions and the workaround solutions they employ reduce the 
-possibility of using the components of the invention for other purposes. Specifically, the image 
sensor of these inventions cannot be utilized for other applications. In contrast, the image sensor of 
the current invention may be any conventional video conferencing camera and may be used as such 
while simultaneously serving this invention. A case in point is the way Tegrity's product uses the 
same image sensor to drive various functions of the product, including image capture, recognition 
of human touching of "virtual buttons," and other functions. Modern projector technology create 
challenges that cannot be met by the prior art, but rather require more extensive use of the available 
information to produce consistent results that can be reliably repeated in a wide variety of useful 
configurations. 

piq^l DESCRIPTION of the preferred EMBODIMENT OF THE INVENTION 
Components 
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A goal of the invention is to be flexible and inexpensive by utilizing commonly available, 
"off the shelf hardware components as much as possible. These include: 

1 . A whiteboard, screen or other Surface [1 1 ]. 

2. An Image Sensor [12] such as a video-conferencing type video camera aimed at the 
Surface. For example, the Tegrity system is sold with an NCK41CV manufactured by 
Howard Enterprises Inc. of Camarillo, CA (a 450 TV-line, NTSC video camera). A 
frame-grabber card may be required to convert the video information to a digital image 
inside the computer. For example, the Tegrity system uses video capture card C210A 
from Tekram Technology, Taipei, Taiwan. Conventional cabling connects the image 
sensor to the computer. 

3. A computer [13]. This is a conventional personal computer, such as an Intel Pentium 
series computer. 

4. A pointing device [14]. The preferred embodiment of the invention employs a standard 
laser-pointer modified for the purposes of the invention as shown in Figure 3. For 
example the "Ultra Infinter" by Quarton Inc. is one of many available models that may 
be used. The modification entails adding a small, black foam ball (30 milimeter 
diameter) near the tip of the laser-pointer. The ball is useful in that it facilitates more 
accurate recognition of the pointer and provides a basis for determining the direction of 
pointing. Alternative embodiments of the invention may employ other types of pointing 
devices with or without light-emitting capabilities. 

5. A projector [1 5] to project the Computer Display Image onto the Surface. The projector 
is not essential to the operation of the invention, however, the invention will typically be 
more useful when one is employed. Any conventional LCD, DLP or other type of 
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projector may be used. For example Proxima 2710, 5800 or InFocus 425, 725, 735 and 
many others. 

6. Communications interface [16] connecting the computer to some communications 
infrastructure. This component is required only when the invention is employed for 
remote presentations or similar uses. This may comprise, for example, a local area 
network controller or a modem interface. 

Tegrity Inc. will market the laser-pointer device and computer-software that implement the 
preferred embodiment of the invention under the name InterPointer (see the product brochure). 

Usa ge Models 

In the preferred embodiment of the invention the spot of light generated by the laser-pen is 
detected as a "pointing event." Similarly, when the light is moved to another location or when it is 
turned off are each detected as a unique event In addition, rapid sequencing of the light on and off 
multiple times can be detected as yet another set of events. These capabilities are currently used to 
support two distinct usage models for the invention, although additional models and applications are 
easily adapted without significant modifications: 

1. "Mouse Emulation" - the light spot simulates the left mouse button on a standard 
computer mouse device. Appearance of the spot simulates pressing of the left mouse 
button, moving it simulates moving the mouse and disappearance of the spot simulates 
releasing the mouse button. This enables emulating the common mouse commands 
"click" and "drag-and-drop" that act upon the location at which the spot appeared. When 
the light is turned on and off twice in succession the "double-click" command can be 
emulated. 
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2. "Pointing" (e.g. for remote presentations) - when the light spot appears it indicates that 
a cursor should be displayed at the corresponding location (at remote sites). Similarly, 
when it moves, the cursor should be moved and when it disappears so should the cursor 
display. In this model directional information is used so that the cursor may indicate the 
direction in which the user is pointing. 

Overview o * System Operation 

The system obtains images from the Image Sensor [12] at a rate of approximately 30 images 
per second. The large amount of information supplied by the Image Sensor (typically over 1 8 million 
bytes per second) and the need to consume a minimal amount of processing resources dictate a 
.strategy of analysis that is not monolithic. Rather, the analysis is broken into several phases such 
that each phase performs more intensive processing on less information than the prior phase. In 
general, these phases include: 

1. Preliminary Screening 

2. Detection 

3. Verification 

Each processing phase comprises one or more functional modules as described below. An additional 
module, the "Hot-Spot Tracker," is executed at a reduced frequency (twice per second). 

When the system determines that a pointing event has occurred at a specific position in the 
Viewed Image it sends a "system event" or activates a predetermined system function using 
techniques that are specific to the computer operating system in use (for example, the 
"mouse_eventO" in Microsoft Window?). When doing so, the display coordinates of the position 
at which the event should be activated are also provided. These coordinates are obtained by 
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converting the Viewed Image coordinates to Computer Display Image coordinates by means of the 
Warping transformation. This transformation is initially computed during a calibration procedure, 
which is explained in detail in the documents incorporated herein by reference. 

In order to use computing resources in an optimal manner the system operates as a "finite 
state machine." This means that at any given time the system will be in one of several "states." 
Specifically, the states currently used are "off," "suspected," and "on." Each state defines what 
analysis will be performed on the next image as well as the possible states that the system may enter 
as a result of this analysis. Specifically, the Preliminary Screening phase of processing is 
considered necessary only when the system is "off," i.e. when no pointing activity has been taking 
place. Once pointing is "suspected" (after Prelminary Screening determines that there may be light 
.spots in the Viewed Image) this phase is not necessary because the image regions where light spots 
may occur will not differ from one processing to the next. When the system has determined with 
certainty that a pointing event is taking place (after several successive cycles so as to verify this 
decision over time) it switches to "on" state. Similarly, when the system determines there is no 
activity it will switch back from "on" to "suspected" and from there to "off." 

We now describe the processing phases listed above in further detail. 

Preliminary Screening 

The system must detect the activation of the pointing device in "real-time," i.e. fast enough 
to follow typical human activity. In the preferred embodiment of the invention this involves 
detecting the spot of light produced by a laser-pointer device and determining if it has been turned 
on and off once or even twice. The latter case would typically be useful for supporting the 
simulation of mouse "double-click" events. This requires the system to process a large amount of 
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information (640x480 pixels for a typical camera) every 33 milliseconds. Full analysis of so much 
information is not practical using state-of-the-art personal computers given that this invention must 
coexist with other applications being driven by the computer without noticeably impacting their 
performance. The solution to this problem is to reduce the amount of information and complexity 
of processing in this preliminary phase. The system performs a fast preliminary screening to detect 
small areas ("regions of interest") that are suspected of containing a laser's spot of light. In later 
processing of this image-cycle, the system will "focus" only on these small regions of interest 
(typically 60x60 image-pixels). Figure 4 shows a small image region that demonstrates how 
Preliminary Screening works. Figure 4(a) shows a pixel image (without color) of a region in which 
the laser device [41] is activated. The foam ball used by this embodiment of the invention can be 
seen [42] as can the hand of the user [43]. The spot of light that the laser device produces is readily 
visible [44]. 

Preliminary screening works in the following manner. The Viewed Image is scanned for 
pixels that have an intensity value above a near-white threshold. Scanning all pixels is time 
consuming, therefore the process skips over most pixels, sampling only 1 out of every 16 pixels. 
This may be done without missing light spots because the minimal size of the spot is 4x4 pixels for 
all practical applications (using a 640x480 pixel Viewed Image). Figure 4(b) and 4(c) demonstrate 
how this process operates on the image region of Figure 4(a). In Figure 4(b) each black dot 
represents a pixel coordinate position at which the screening process will sample the value of the 
corresponding pixel from the image of Figure 4(a). This selective sampling produces the image of 
Figure 4(c). This image contains 1/16* of the information contained in the image of Figure 4(a) 
however the spot of laser light is still readily apparent at [46]. 
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The sampled pixels that are bright enough (high intensity) and are in close proximity to each 
other are subsequently treated as a single region of interest with a suspected light-spot. In the case 
of Figure 4, the bright spot of [46] will be identified as such a suspected light spot. Note that Figure 
4(a) shows only a small image area, which is, in feet, a "suspected region " In practice, Preliminary 
Screening is used on the entire Viewed Image precisely in order to locate and identify such regions 
in the much larger image. 

The process of preliminary screening ignores certain image-pixels and does not add them to 
suspected regions even if they exceed the intensity threshold. The set of ignored pixels is determined 
by the Hot-Spot Tracking module as described below. If no suspected regions are found, processing 
is complete for this cycle and the system may change its state accordingly. 

Detection 

This module recognizes a pattern of a spot of laser-light hitting the Surface. It can detect any 
kind of lasers (635nm to 650nm wavelengths). Surprisingly a laser-dot as captured by a typical video 
camera is not red. Actually it appears as a white spot due to the high intensity of the laser. This 
module operates on predefined regions of interest as follows: It thresholds the image with a high 
threshold, keeping only pixels of high intensity. The binarized image is then segmented into 
connected blobs. Then the algorithm looks for a blob that best matches the pattern of an expected 
spot of light in its dimensions, measure of roundness. These steps use techniques that are well known 
in the art. The end result is that either no blobs are determined to match the expected characteristics 
of a light spot or a single blob is selected. In the former case processing is complete with negative 
results (no activation) and the system may change state accordingly. 
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Verification 

Verification is necessary to prevent false triggering caused by reflective objects like watches 
and jewelry or by saturation of small regions of the Viewed linage due to the Image Sensor's 
dynamic-range. The latter is commonly seen when presenting the Image Sensor with a display 
containing high contrast (typically with Automatic Gain Compensation settings used by most video 
cameras). A major drawback of similar devices introduced in the past was the lack of such 
verification procedures and relying solely on an intensity threshold to conclude whether or not 

pointing was activated. 

Two distinct kinds of verification may be used in the preferred embodiment of the invention: 

1 . Fiducial-based Verification: This method is used to recognize an a priori known figure 
or pattern known as a "fiducial". In our case the fiducial is the black foam ball attached 
to the laser-pointer device as shown in Figure 3. The ball is detected as follows. A small 
region of interest in the Viewed Image (about 60x60 pixels) surrounding the detected 
spot is analyzed. In this region the algorithm looks for the ball using a simple 
"correlation" with an idealized ball. If the correlation indeed finds a ball, the image is 
binarized by its intensity, using a threshold that is determined during the correlation 
check (utilizing the knowledge that the ball is black). The "blob" that defines the ball's 
pixels is checked to be round and of fitting dimensions and proportions. Once the system 
locates the fiducial the direction in which the user is pointing is easily determined. 

2. Skin-based Verification: This method is used to verify that human skin is apparent in the 
Viewed Image in close proximity to the spot at which light was detected This is based 
on the assumption that when activating the pointer the user's hand is near the light spot 
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and is visible in the Viewed Image. The verification procedure is based on a "skin 
detector" function, which detects pixels that have a high likelihood of corresponding to 
human skin. The algorithm uses a statistical model of human-skin color and is based on 
current research techniques in this field (see "Visual Tracking for Multimodal Human 
Computer Interaction" - Jie Yang, Rainer Stiefelhagen, Uwe Meier, Alex Waibel at 
CHI98). 

^r.t-dpot Tracking 

One problem that the system must deal with is the presence of "hot-spots" in the Viewed 
Image. Hot-spots are regions of the image that appear saturated, i.e. where pixels are assigned the 
-highest possible intensity values (white) due to the relative contrast of that area with other areas of 
the viewed Surface. For example, the reflection of a projector's beam, which may be blinding to a 
viewer, will typically produce this effect in a digital image. Similarly, in some cases projecting a 
highly contrasting image (such as one with a dark background and a small amount of bright pixels) 
may produce hot-spots. In these areas differences of intensity cannot be distinguished, therefore the 
light-spot of a laser will not be detected within them. The main problem is that hot-spots may 
sometimes appear similar in characteristics (size, shape, intensity) to typical light spots, thus 
generating false triggering of this invention. 

The Hot-Spot Tracking module solves this problem by continuously tracking and "learning'' 
which pixels belong to hot-spots. These pixels are remembered by saving a "mask" which represents 
locations to be ignored by the Preliminary Screening phase described above. The idea is to compute 
and maintain a running average image of the last few seconds and threshold it to produce a binary 
mask. Pixels that are steadily saturated will quickly be marked in this mask and will be ignored in 
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subsequent processing cycles. If and when such pixels cease to be saturated (e.g., the projected 
image changes or room lighting changes), the system will quickly adapt to the change and the pixels 
be no longer be marked in the mask, hence they will again participate in the analysis. 

This module is activated at a low fiequency - approximately twice per second. This rate 
(along with the averaging coefficient used for computing the running average) is quick enough to 
prevent false triggering by hot-spots yet slow enough to have a negligible impact on performance. 
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The Tegrity InterPointer 

The Tegrity InterPointer can be used in place of a mouse, to draw elements or make menu 
selections directly on the white board, and as a pointing device that can be broadcast 
during WebLeamer distance learning. 

To use the InterPointer, the InterPointer Add-on software must be installed, and a serial 
number entered. This is done through the installation program on the Tegrity Digital 
Flipchart CD. See your Tegrity Digital Flipchart User's Guide, chapter 4, for installation 
instructions. 

Please remember to insert the 2 (included) batteries into the pointer according to the 
instructions. 

Once the InterPointer software is installed it must be enabled and activated. 

Enabling the InterPointer for Mouse Emulation 

O Select Options from the Tegrity pull down menu and bring the General tab 

forward. Click on the Enable InterPointer option. 
© Access the Tegrity Touch Panel 
© Right click on the Touch Panel and select InterPointer. 

Or 

pa x! Use the InterPointer to click on the able/disable button on the top of the Touch Panel. 
A checkmark displays beside the option in the panel. 

You can use the InterPointer to control any buttons or menus, from the whiteboard. 

Enabling the InterPointer for the WebLearner 

• Select Options from the Tegrity pull down menu and bring the General tab 
forward. Click on the Enable InterPointer option. 

From now on the InterPointer will be activated automatically each time you work with 
WebLearner. 

Wherever you point with the InterPointer on the whiteboard, the student sees a small 
hand in the same location on the web class page. This hand is streamed with the live 
class and is saved with the recorded class file. This way you can show your students 
specific points on the whiteboard while you talk about them during your class. 

Using the InterPointer 

Warning: Avoid direct eye exposure! Never look into the ray emitted by the InterPointer. 



O Keep the InterPointer as close to the 
whiteboard as possible. The tip of the 
InterPointer should almost be touching the 
board. In order for the camera to see the 
InterPointer, the "ball" must be inside the 
projected area. 

6 Press down on the Clip while pointing 
with the InterPointer, as shown in the 
figure beside. 



HoU the power Wb? " 

doss to the object or ~Vv- w ^. 

menu torn (a 
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WHAT IS CLAIMED IS: 

1 . A method of controlling the display, transfer and processing of information in 

computer systems, said method comprising the step of a user pointing at a wll surface 
which contains information to activate predetermined operations relating to that 
information. 
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